Information Retrieval and Information Theory

Information retrieval is concerned with how to classify information and how to judge the similarity between two objects, such aswritten documents. As the amount of information available in digital formhas grown, so too has theneed for accurate and scalable algorithms for handling this information. Information theory is concerned with the production and transmission of information. Using a framework known as the source-channel model of communication, information theory has established theoretical bounds on the limits of data compression and communication in the presence of noise and has contributed to practical technologies as varied as cellular communication and statistical translation. In this work we establish a set of connections between information retrieval and information theory. In particular, we focus on two central and well-studied problems in retrieval: classifying documents and ranking documents by relevance to a query. Viewing these problems from a source-channel perspective, we develop new statistical algorithms for each: to the classification problem we apply methods from errorcorrecting coding theory, and to the ranking problem we apply methods from statistical translation. In both cases, we report on the architecture and empirical behavior of prototype systems, whose performance improves on that of state of the art techniques. 1 Two Problems in Information Retrieval